Accommodating for 3D Head Movement in Visual Lipreading

In automatic lipreading, the speaker’s head movement can affect the mouth shape appearing in the captured images independently of the true mouth shape. Such distortion can lead to incorrect recognition of visual speech, thus this problem must be dealt with for a practical application. We have developed a system that accomodates the 3D head movement of the speaker in recognising the mouth shape of unadorned lips appearing in the image. This is achieved by tracking the 3D head movement from the 2D input and correcting the mouth shape that is detected from the image, in order to robustly recognise the visual phonemes. The system allows the speaker’s head to move and exhibit rotations of up to 30 degrees away from the camera. An experiment is presented, showing head movements in 3D while the speaker enunciates 5 phonemes that represent the various mouth shapes of visual speech. The result shows that the 3D head tracker robustly tracks the 3D head movement, and accomodating head pose to correct mouth shape is achieved effectively.